Red Hat Enterprise Linux 7 Troubleshooting

Networking Issues

Module Topics

Basic Network Configuration
Packet Monitoring
Network Performance Monitoring
Network Performance Configuration
MTU and Jumbo Frames
iSCSI Troubleshooting

Packet Monitoring

Packet Monitoring Tools

Use tcpdump to do narrow search based on several parameters.
```
[root@server1 ~]# tcpdump -i eth0 -n src host instructor.example.com and src port 53
  ... output omitted ...
```
This reports all packets from instructor.example.com originating from port 53.
Use nc to set up a client/server chat:
1. On one host, start nc with the -l (listen) option and a port number.
  [student@server1 ~]$ nc -l 5150
2. On another system, start a client process to connect to the port on which the server is listening.
  [student@desktop1 ~]$ nc server1 5150
  Anything typed on the keyboard is sent across the connection between the server and client.
You can also use tshark, provided by the wireshark package, to capture traffic to a file.
```
[root@desktop1 ~]# tshark -f "tcp port 5150" -i br0 -w /tmp/capture.cap
```
To interpret or explore the captured content in a graphical tool, run wireshark.
```
[root@desktop1 ~]# wireshark /tmp/capture.cap
```

Use nc in conjunction with tcpdump or tshark, to troubleshoot connections between machines.

Reference

tcpdump(8), nc(1), tshark(1), and wireshark(1) man pages

== m10p02_packet_monitor

In this section you will learn how to monitor packets.

In situations where services appear to be available but do not work fully, or worse, hang under certain conditions, packet monitoring tools can be utilized in order to follow a conversation to determine the fault. Both tcpdump and tshark are command line tools for packet monitoring, and wireshark is a GUI tool for tshark.

One key to success when using packet monitoring tools is to have a very specific search. So much information is passed back and forth between machines that, without a narrow criterion, the information you seek may be lost in a deluge of other packets. tcpdump allows you to narrow the search on the basis of a large number of parameters.
This reports all packets from instructor.example.com originating from port 53.
Another useful tool for debugging applications is nc (netcat). Netcat can be used for just about anything involving TCP and UDP packets. You can set it up to listen for conversations or initialized TCP connections, even UDP send packets. The simplest way to use nc is to set up a client/server chat. On one host, start nc with the -l (listen) option and a port number.
Then on another system start up a client process to connect to the port the server is listening on.

Anything typed in on the keyboard is sent across the connection between the server and client. This is all done in clear text, so capturing this exchange is trivial.
tshark, provided by the wireshark, package can also be used to capture the traffic to a file.
To interpret or explore the resulting captured content in a graphical tool, you can run wireshark.

Using nc in conjunction with tcpdump or tshark, you can troubleshoot connections between machines.

Network Performance Monitoring

A packet sent to a Red Hat Enterprise Linux system is received by the network interface card (NIC) and placed in either an internal hardware buffer or a ring buffer.
The NIC sends a hardware interrupt request, prompting creation of a software interrupt operation to handle the interrupt request.
The packet is transferred from the buffer to the network stack.
Depending on the packet and network configuration, the packet is forwarded, discarded, or passed to a socket receive queue for an application.
The packet is removed from the network stack.

== m10p03_monitor_diagnose_network_1

In this section you will learn about tools used to monitor and diagnose network performance.

To make good tuning decisions, you need a thorough understanding of packet reception in Red Hat Enterprise Linux. This section explains how network packets are received and processed, and where potential bottlenecks can occur.

A packet sent to a Red Hat Enterprise Linux system is received by the network interface card (NIC) and placed in either an internal hardware buffer or a ring buffer. The NIC then sends a hardware interrupt request, prompting the creation of a software interrupt operation to handle the interrupt request.

As part of the software interrupt operation, the packet is transferred from the buffer to the network stack. Depending on the packet and your network configuration, the packet is then forwarded, discarded, or passed to a socket receive queue for an application and then removed from the network stack. This process continues until either there are no packets left in the NIC hardware buffer, or a certain number of packets (specified in /proc/sys/net/core/dev_weight) are transferred.

Network performance problems are most often the result of hardware malfunction or faulty infrastructure. Red Hat highly recommends verifying that your hardware and infrastructure are working as expected before beginning to tune the network stack.

Network Performance Monitoring

There are a number of points during network packet processing that can become bottlenecks and reduce performance:

The NIC hardware buffer or ring buffer
- The hardware buffer might be a bottleneck if a large number of packets are being dropped.
The hardware or software interrupt queues
- Interrupts can increase latency and processor contention.
The socket receive queue for the application
- A bottleneck in an application’s receive queue is indicated by a large number of packets that are not copied to the requesting application, or by an increase in UDP input errors (InErrors) in /proc/net/snmp.

Network Performance Monitoring

Diagnostic Tools

ss is a command-line utility that prints statistical information about sockets.
- Allows administrators to assess device performance over time.
- By default, ss lists open non-listening TCP sockets that have established connections.
- Options are provided to filter statistics about specific sockets.
- Red Hat recommends ss over netstat in Red Hat Enterprise Linux 7.
- ss is provided by the iproute package.
ip is a utility that lets administrators manage and monitor routes, devices, routing policies, and tunnels.
- ip monitor can continuously monitor the state of devices, addresses, and routes.
- ip is provided by the iproute package.
dropwatch is an interactive tool that monitors and records packets that are dropped by the kernel.
ethtool is a utility that allows administrators to view and edit network interface card settings.
- Useful for observing statistics of certain devices, such as number of packets dropped by that device.
- View the status of a specified device’s counters with ethtool -S and the name of the device you want to monitor.
  [root@server1 ~]# ethtool -S devname
/proc/net/snmp displays data used by snmp agents for IP, ICMP, TCP and UDP monitoring and management.
- Examine this file on a regular basis to identify unusual values and potential performance problems.
- An increase in UDP input errors (InErrors) in /proc/net/snmp can indicate a bottleneck in a socket receive queue.

Reference

ss, ip, dropwatch, and ethtool man pages

== m10p05_monitor_tools_1

Red Hat Enterprise Linux 7 provides a number of tools that are useful for monitoring system performance and diagnosing performance problems related to the networking subsystem. This section outlines the available tools and gives examples of how to use them to monitor and diagnose network related performance issues.

ss is a command-line utility that prints statistical information about sockets, allowing administrators to assess device performance over time. By default, ss lists open non-listening TCP sockets that have established connections, but a number of useful options are provided to help administrators filter out statistics about specific sockets.
Red Hat recommends ss over netstat in Red Hat Enterprise Linux 7.

ss is provided by the iproute package. For more information, see the ss man page.
The ip utility lets administrators manage and monitor routes, devices, routing policies, and tunnels. The ip monitor command can continuously monitor the state of devices, addresses, and routes.
ip is provided by the iproute package. For more information, see the ip man page.
dropwatch is an interactive tool that monitors and records packets that are dropped by the kernel.
For more information, see the dropwatch man page.
The ethtool utility allows administrators to view and edit network interface card settings. It is useful for observing the statistics of certain devices, such as the number of packets dropped by that device.
You can view the status of a specified device’s counters with ethtool -S and the name of the device you want to monitor.

For more information, see the ethtool man page.
The /proc/net/snmp file displays data that is used by snmp agents for IP, ICMP, TCP and UDP monitoring and management. Examining this file on a regular basis can help administrators identify unusual values and thereby identify potential performance problems. For example, an increase in UDP input errors (InErrors) in /proc/net/snmp can indicate a bottleneck in a socket receive queue.

Network Performance Monitoring

Example SystemTap Scripts

nettop.stp prints a list of processes (process identifier and command) every 5 seconds. The list includes the number of packets sent/received and the amount of data sent/received by the process during that interval.
socket-trace.stp instruments each function in the Linux kernel’s net/socket.c file and prints trace data.
tcp_connections.stp prints information for each new incoming TCP connection accepted by the system, including:
- UID
- Command accepting the connection
- Process identifier of the command
- Port the connection is on
- IP address of the request’s originator
dropwatch.stp prints, at 5 second intervals, the number of socket buffers freed at locations in the kernel.

Reference

Red Hat Enterprise Linux 7 SystemTap Beginner's Guide

== m10p06_monitor_tools_2

The Red Hat Enterprise Linux 7 SystemTap Beginner’s Guide includes several sample scripts that are useful for profiling and monitoring network performance.

The following SystemTap example scripts relate to networking and may be useful in diagnosing network performance problems. By default they are installed to the /usr/share/doc/systemtap-client/examples/network directory.

Every 5 seconds, nettop.stp prints a list of processes (process identifier and command) with the number of packets sent and received, and the amount of data sent and received, by the process during that interval.

socket-trace.stp instruments each of the functions in the Linux kernel’s net/socket.c file, and prints trace data.

tcp_connections.stp prints information for each new incoming TCP connection accepted by the system. The information includes the UID, the command accepting the connection, the process identifier of the command, the port the connection is on, and the IP address of the originator of the request.

Every 5 seconds, dropwatch.stp prints the number of socket buffers freed at locations in the kernel.

The Red Hat Enterprise Linux 7 SystemTap Beginner’s Guide is available from http://access.redhat.com/site/documentation/Red_Hat_Enterprise_Linux/

Network Performance Configuration

Tuned-adm Profiles for Network Performance

tuned-adm is a command line tool that provides profiles to improve performance in a number of specific use cases. The following profiles can be useful for improving networking performance:
- latency-performance
- network-latency
- network-throughput
tuned-adm recommend is a sub-command that assesses your system and outputs a recommended tuning profile.
- Also sets the default profile for your system at install time.
- Can be used to return to the default profile.
As of Red Hat Enterprise Linux 7, tuned-adm can run any command as part of enabling or disabling a tuning profile.
- Use to add environment-specific checks that are not available in tuned-adm, such as checking whether the system is the master database node before selecting which tuning profile to apply.
- include parameter in profile definition files allows you to base your own tuned-adm profiles on existing profiles.

== m10p07_network_ref_tuned_adm_1

In this section you will learn about tools and kernel parameters that you can configure to tune network performance.

Red Hat Enterprise Linux provides a number of tools to assist administrators in configuring the system. This section outlines the available tools and provides examples of how they can be used to solve network related performance problems in Red Hat Enterprise Linux 7.

However, it is important to keep in mind that network performance problems are sometimes the result of hardware malfunction or faulty infrastructure. Red Hat highly recommends verifying that your hardware and infrastructure are working as expected before using these tools to tune the network stack.

Further, some network performance problems are better resolved by altering the application than by reconfiguring your network subsystem. It is generally a good idea to configure your application to perform frequent POSIX calls, even if this means queuing data in the application space, as this allows data to be stored flexibly and swapped in or out of memory as required.

tuned-adm is a command line tool that provides a number of different profiles to improve performance in a number of specific use cases. The following profiles can be useful for improving networking performance.

latency-performance
network-latency
network-throughput

tuned-adm also provides a sub-command (tuned-adm recommend) that assesses your system and outputs a recommended tuning profile. This also sets the default profile for your system at install time, so it can be used to return to the default profile.

As of Red Hat Enterprise Linux 7, tuned-adm includes the ability to run any command as part of enabling or disabling a tuning profile. This allows you to add environment-specific checks that are not available in tuned-adm, such as checking whether the system is the master database node before selecting which tuning profile to apply.

Red Hat Enterprise Linux 7 also provides the include parameter in profile definition files, which allows you to base your own tuned-adm profiles on existing profiles.

Network Performance Configuration

Tuning Profiles Provided with tuned-adm and Supported in Red Hat Enterprise Linux 7

throughput-performance - A server profile focused on improving throughput.
- Default profile recommended for most systems.
- Favors performance over power savings by setting intel_pstate and max_perf_pct=100.
- Enables transparent huge pages.
- Uses cpupower to set the performance cpufreq governor.
- Sets the input/output scheduler to deadline.
- Sets kernel.sched_min_granularity_ns to 10 µs, kernel.sched_wakeup_granularity_ns to 15 µs, and vm.dirty_ratio to 40%.
latency-performance - A server profile focused on lowering latency.
- Recommended for latency-sensitive workloads that benefit from c-state tuning and increased TLB efficiency of transparent huge pages.
- Favors performance over power savings by setting intel_pstate and max_perf_pct=100.
- Enables transparent huge pages.
- Uses cpupower to set the performance cpufreq governor.
- Requests a cpu_dma_latency value of 1.

== m10p08_network_ref_tuned_adm_2

The following tuning profiles are provided with tuned-adm and are supported in Red Hat Enterprise Linux 7.

throughput-performance - A server profile focused on improving throughput. This is the default profile, and is recommended for most systems. This profile favors performance over power savings by setting intel_pstate and max_perf_pct=100. It enables transparent huge pages, uses cpupower to set the performance cpufreq governor, and sets the input/output scheduler to deadline. It also sets kernel.sched_min_granularity_ns to 10 µs, kernel.sched_wakeup_granularity_ns to 15 µs, and vm.dirty_ratio to 40%.
latency-performance - A server profile focused on lowering latency. This profile is recommended for latency-sensitive workloads that benefit from c-state tuning and the increased TLB efficiency of transparent huge pages. This profile favors performance over power savings by setting intel_pstate and max_perf_pct=100. It enables transparent huge pages, uses cpupower to set the performance cpufreq governor, and requests a cpu_dma_latency value of 1.

Network Performance Configuration

Tuning Profiles Provided with tuned-adm and Supported in Red Hat Enterprise Linux 7

network-latency - A server profile focused on lowering network latency.
- Favors performance over power savings by setting intel_pstate and max_perf_pct=100.
- Disables transparent huge pages and automatic NUMA balancing.
- Uses cpupower to set the performance cpufreq governor.
- Requests a cpu_dma_latency value of 1.
- Sets busy_read and busy_poll times to 50 µs, and tcp_fastopen to 3.
network-throughput - A server profile focused on improving network throughput.
- Favors performance over power savings by setting intel_pstate and max_perf_pct=100 and increasing kernel network buffer sizes.
- Enables transparent huge pages.
- Uses cpupower to set the performance cpufreq governor.
- Sets kernel.sched_min_granularity_ns to 10 µs, kernel.sched_wakeup_granularity_ns to 15 µs, and vm.dirty_ratio to 40%.

== m10p09_network_ref_tuned_adm_3

network-latency - A server profile focused on lowering network latency. This profile favors performance over power savings by setting intel_pstate and max_perf_pct=100. It disables transparent huge pages, and automatic NUMA balancing. It also uses cpupower to set the performance cpufreq governor, and requests a cpu_dma_latency value of 1. It also sets busy_read and busy_poll times to 50 µs, and tcp_fastopen to 3.
network-throughput - A server profile focused on improving network throughput. This profile favors performance over power savings by setting intel_pstate and max_perf_pct=100 and increasing kernel network buffer sizes. It enables transparent huge pages, and uses cpupower to set the performance cpufreq governor. It also sets kernel.sched_min_granularity_ns to 10 µs, kernel.sched_wakeup_granularity_ns to 15 µs, and vm.dirty_ratio to 40%.

Network Performance Configuration

Tuning Profiles Provided with tuned-adm and Supported in Red Hat Enterprise Linux 7

virtual-guest - A profile focused on optimizing performance in Red Hat Enterprise Linux 7 virtual machines.
- Favors performance over power savings by setting intel_pstate and max_perf_pct=100
- Decreases swappiness of virtual memory
- Enables transparent huge pages
- Uses cpupower to set the performance cpufreq governor
- Sets kernel.sched_min_granularity_ns to 10 µs, kernel.sched_wakeup_granularity_ns to 15 µs, and vm.dirty_ratio to 40%
virtual-host - A profile focused on optimizing performance in Red Hat Enterprise Linux 7 virtualization hosts.
- Favors performance over power savings by setting intel_pstate and max_perf_pct=100
- Decreases swappiness of virtual memory
- Enables transparent huge pages
- Writes dirty pages back to disk more frequently
- Uses cpupower to set the performance cpufreq governor
- Sets kernel.sched_min_granularity_ns to 10 µs, kernel.sched_wakeup_granularity_ns to 15 µs, kernel.sched_migration_cost to 5 µs, and vm.dirty_ratio to 40%.

References

Red Hat Enterprise Linux 7 Power Management Guide
man tuned-adm man page

== m10p10_network_ref_tuned_adm_4

virtual-guest - A profile focused on optimizing performance in Red Hat Enterprise Linux 7 virtual machines. This profile favors performance over power savings by setting intel_pstate and max_perf_pct=100. It also decreases the swappiness of virtual memory. It enables transparent huge pages, and uses cpupower to set the performance cpufreq governor. It also sets kernel.sched_min_granularity_ns to 10 µs, kernel.sched_wakeup_granularity_ns to 15 µs, and vm.dirty_ratio to 40%.
virtual-host - A profile focused on optimizing performance in Red Hat Enterprise Linux 7 virtualization hosts. This profile favors performance over power savings by setting intel_pstate and max_perf_pct=100. It also decreases the swappiness of virtual memory. This profile enables transparent huge pages and writes dirty pages back to disk more frequently. It uses cpupower to set the performance cpufreq governor. It also sets kernel.sched_min_granularity_ns to 10 µs, kernel.sched_wakeup_granularity_ns to 15 µs, kernel.sched_migration_cost to 5 µs, and vm.dirty_ratio to 40%.

For detailed information about the power saving profiles provided with tuned-adm, see the Red Hat Enterprise Linux 7 Power Management Guide, available from http://access.redhat.com/site/documentation/Red_Hat_Enterprise_Linux/

For detailed information about using tuned-adm, see the man page: man tuned-adm.

Network Performance Configuration

Configuring the Hardware Buffer

Slow the input traffic
- Filter incoming traffic
- Decrease the rate at which the queue fills
  - Reduce the number of joined multicast groups
  - Reduce the amount of broadcast traffic
Resize the hardware buffer queue
- Increase queue size, so it does not easily overflow
  - Reduce the number of packets being dropped
- Modify the rx/tx parameters of the network device with ethtool:
  [root@server1 ~]# ethtool --set-ring devname value
Change the drain rate of the queue
- Device weight or dev_weight is the number of packets a device can receive in a single scheduled processor access
- Increase the rate at which a queue drains by increasing its dev_weight
  - Temporarily alter dev_weight by changing the contents of /proc/sys/net/core/dev_weight
  - Permanently alter dev_weight with sysctl; provided by procps-ng

Reference

Red Hat Enterprise Linux System Administrator's Guide

== m10p11_network_ref_hdwr_buffer

If a large number of packets are being dropped by the hardware buffer, there are a number of potential solutions.

Slow the input traffic - Filter incoming traffic, reduce the number of joined multicast groups, or reduce the amount of broadcast traffic to decrease the rate at which the queue fills. For details of how to filter incoming traffic, see the Red Hat Enterprise Linux 7 Security Guide.

For details about multicast groups, see the Red Hat Enterprise Linux 7 Clustering documentation. For details about broadcast traffic, see the Red Hat Enterprise Linux 7 System Administrator’s Reference Guide, or documentation related to the device you want to configure. All Red Hat Enterprise Linux 7 documentation is available from http://access.redhat.com/site/documentation/Red_Hat_Enterprise_Linux/

Resize the hardware buffer queue - Reduce the number of packets being dropped by increasing the size of the queue so that it does not overflow as easily. You can modify the rx/tx parameters of the network device with the ethtool command:
Change the drain rate of the queue - Device weight refers to the number of packets a device can receive at one time (in a single scheduled processor access). You can increase the rate at which a queue is drained by increasing its device weight, which is controlled by the dev_weight parameter. This parameter can be temporarily altered by changing the contents of the /proc/sys/net/core/dev_weight file, or permanently altered with sysctl, which is provided by the procps-ng package.
Altering the drain rate of a queue is usually the simplest way to mitigate poor network performance. However, increasing the number of packets that a device can receive at one time uses additional processor time, during which no other processes can be scheduled, so this can cause other performance problems.

Network Performance Configuration

Configuring Busy Polling

To enable busy polling on specific sockets:

Set sysctl.net.core.busy_poll to a value other than 0.
- Number of microseconds to wait for packets on the device queue for socket poll and selects
- Red Hat recommends a value of 50
Add the SO_BUSY_POLL socket option.

To enable busy polling globally:

Set sysctl.net.core.busy_read to a value other than 0.
- Number of microseconds to wait for packets on the device queue for socket reads
- Sets default value of SO_BUSY_POLL.
- Red Hat recommends a value of 50 for a small number of sockets, 100 for large numbers of sockets
- For more than several hundred sockets, use epoll instead.

Busy polling behavior is supported by the following drivers, which are also supported on Red Hat Enterprise Linux 7:

bnx2x
be2net
ixgbe
mlx4
myri10ge

== m10p12_network_ref_busy_polling

If analysis reveals high latency, your system may benefit from poll-based, rather than interrupt-based, packet receipt.

Busy polling helps reduce latency in the network receive path by allowing socket layer code to poll the receive queue of a network device, and disabling network interrupts. This removes delays caused by the interrupt and the resultant context switch. However, it also increases CPU utilization. Busy polling also prevents the CPU from sleeping, which can incur additional power consumption.

Busy polling is disabled by default. To enable busy polling on specific sockets, do the following:

Set sysctl.net.core.busy_poll to a value other than 0. This parameter controls the number of microseconds to wait for packets on the device queue for socket poll and selects. Red Hat recommends a value of 50.
Add the SO_BUSY_POLL socket option to the socket.
To enable busy polling globally, you must also set sysctl.net.core.busy_read to a value other than 0. This parameter controls the number of microseconds to wait for packets on the device queue for socket reads. It also sets the default value of the SO_BUSY_POLL option. Red Hat recommends a value of 50 for a small number of sockets, and a value of 100 for large numbers of sockets. For extremely large numbers of sockets (more than several hundred), use epoll instead.

Busy polling behavior is supported by the following drivers. These drivers are also supported on Red Hat Enterprise Linux 7.
bnx2x
be2net
ixgbe
mlx4
myri10ge

Network Performance Configuration

Configuring Socket Receive Queues

Decrease speed of incoming traffic
- Decrease rate at which queue fills
- Filter or drop packets before they reach the queue
- Lower weight of device
Increase depth of the application’s socket queue
- Match size of traffic bursts to prevent packets from being dropped
Decrease speed of incoming traffic
- Filter incoming traffic
- Lower the network interface card’s device weight

Reference

Red Hat Enterprise Linux Security Guide

== m10p13_network_ref_socket_rcv_queues

If analysis suggests that packets are being dropped because the drain rate of a socket queue is too slow, there are several ways to alleviate the performance issues that result.

Decrease the speed of incoming traffic - Decrease the rate at which the queue fills by filtering or dropping packets before they reach the queue, or by lowering the weight of the device.
Increase the depth of the application’s socket queue - If a socket queue receives a limited amount of traffic in bursts, then increasing the depth of the socket queue to match the size of the bursts of traffic may prevent packets from being dropped.
Decrease the speed of incoming traffic - Filter incoming traffic or lower the network interface card’s device weight to slow incoming traffic. For details of how to filter incoming traffic, see the Red Hat Enterprise Linux 7 Security Guide, available from http://access.redhat.com/site/documentation/Red_Hat_Enterprise_Linux/.
Device weight refers to the number of packets a device can receive at one time (in a single scheduled processor access). Device weight is controlled by the dev_weight parameter. This parameter can be temporarily altered by changing the contents of the /proc/sys/net/core/dev_weight file, or permanently altered with sysctl, which is provided by the procps-ng package.

Network Performance Configuration

Increasing Queue Depth

To increase the depth of a queue, increase the size of the socket receive buffer by making either of the following changes:

Increase the value of /proc/sys/net/core/rmem_default
- Controls default size of receive buffer used by sockets
- Value must be smaller than value of /proc/sys/net/core/rmem_max
Use setsockopt to configure a larger SO_RCVBUF value
- Controls maximum bytes of a socket’s receive buffer
- Use getsockopt system call to determine current buffer value

== m10p14_network_ref_increase_q_depth

Increasing the depth of an application socket queue is typically the easiest way to improve the drain rate of a socket queue, but it is unlikely to be a long-term solution.

To increase the depth of a queue, increase the size of the socket receive buffer by making either of the following changes:

Increase the value of /proc/sys/net/core/rmem_default - This parameter controls the default size of the receive buffer used by sockets. This value must be smaller than that of /proc/sys/net/core/rmem_max.
Use setsockopt to configure a larger SO_RCVBUF value - This parameter controls the maximum size in bytes of a socket’s receive buffer. Use the getsockopt system call to determine the current value of the buffer.

For further information about this parameter, see the man page: man 7 socket.

Network Performance Configuration

Configuring Receive-Side Scaling (RSS)

Receive-Side Scaling (RSS), also known as multi-queue receive
Distributes network receive processing across several hardware-based receive queues
Allows inbound network traffic to be processed by multiple CPUs
Relieves bottlenecks in receive interrupt processing caused by overloading a single CPU
Reduces network latency

Network Performance Configuration

Configuring Receive-Side Scaling (RSS)

To determine whether your NIC supports RSS:

Check whether multiple interrupt request queues are associated with the interface in /proc/interrupts.

Example: p1p1 interface

[root@server1 ~]# egrep 'CPU|p1p1' /proc/interrupts
   CPU0    CPU1    CPU2    CPU3    CPU4    CPU5
89:   40187       0       0       0       0       0   IR-PCI-MSI-edge   p1p1-0
90:       0     790       0       0       0       0   IR-PCI-MSI-edge   p1p1-1
91:       0       0     959       0       0       0   IR-PCI-MSI-edge   p1p1-2
92:       0       0       0    3310       0       0   IR-PCI-MSI-edge   p1p1-3
93:       0       0       0       0     622       0   IR-PCI-MSI-edge   p1p1-4
94:       0       0       0       0       0    2475   IR-PCI-MSI-edge   p1p1-5

Check the output of ls -1 /sys/devices/*/*/device_pci_address/msi_irqs after the network driver is loaded.
Example: Device with PCI address of 0000:01:00.0
```
[root@server1 ~]# ls -1 /sys/devices/*/*/0000:01:00.0/msi_irqs
101
102
103
104
105
106
107
108
109
```

== m10p16_network_ref_config_rss_2

To determine whether your network interface card supports RSS, check whether multiple interrupt request queues are associated with the interface in /proc/interrupts. For example, if you are interested in the p1p1 interface:

The preceding output shows that the NIC driver created 6 receive queues for the p1p1 interface (p1p1-0 through p1p1-5). It also shows how many interrupts were processed by each queue, and which CPU serviced the interrupt. In this case, there are 6 queues because by default, this particular NIC driver creates one queue per CPU, and this system has 6 CPUs. This is a fairly common pattern amongst NIC drivers.

Alternatively, you can check the output of ls -1 /sys/devices/*/*/device_pci_address/msi_irqs after the network driver is loaded. For example, if you are interested in a device with a PCI address of 0000:01:00.0, you can list the interrupt request queues of that device with the following command:

Network Performance Configuration

Configuring Receive-Side Scaling (RSS)

RSS is enabled by default
Configure number of queues for RSS in appropriate network device driver
- For bnx2x driver, configure in num_queues
- For sfc driver, configure in rss_cpus
Typically configured in /sys/class/net/device/queues/rx-queue/
- device = name of network device (such as eth1)
- rx-queue = name of appropriate receive queue
Limit number of queues to one per physical CPU core
RSS distributes network processing equally between available CPUs
- Based on amount of processing each CPU has queued
- To modify and weight activity distribution based on importance, use ethtool --show-rxfh-indir and --set-rxfh-indir
Use irqbalance in conjunction with RSS to reduce likelihood of cross-node memory transfers and cache line bouncing
- Lowers latency of processing network packets

== m10p17_network_ref_config_rss_3

RSS is enabled by default. The number of queues (or the CPUs that should process network activity) for RSS are configured in the appropriate network device driver. For the bnx2x driver, it is configured in num_queues. For the sfc driver, it is configured in the rss_cpus parameter. Regardless, it is typically configured in /sys/class/net/device/queues/rx-queue/, where device is the name of the network device (such as eth1) and rx-queue is the name of the appropriate receive queue.

When configuring RSS, Red Hat recommends limiting the number of queues to one per physical CPU core. Hyper-threads are often represented as separate cores in analysis tools, but configuring queues for all cores including logical cores, such as hyper-threads, has not proven beneficial to network performance.

When enabled, RSS distributes network processing equally between available CPUs based on the amount of processing each CPU has queued. However, you can use the ethtool --show-rxfh-indir and --set-rxfh-indir parameters to modify how network activity is distributed, and to weight certain types of network activity as more important than others.

The irqbalance daemon can be used in conjunction with RSS to reduce the likelihood of cross-node memory transfers and cache line bouncing. This lowers the latency of processing network packets.

Network Performance Configuration

Configuring Receive Packet Steering (RPS)

Advantages of RPS over hardware-based RSS:
- Use RPS with any NIC
- Easy to add software filters to RPS to deal with new protocols
- RPS does not increase hardware interrupt rate of the network device
  - Does introduce inter-processor interrupts
RPS is configured per network device and receive queue
Configure in /sys/class/net/device/queues/rx-queue/rps_cpus
- device = name of network device (such as eth0)
- rx-queue = name of appropriate receive queue (such as rx-0)
Default value of the rps_cpus file is 0
- Disables RPS
- CPU that handles network interrupt also processes packet
Enable RPS by configuring appropriate rps_cpus with CPUs that should process packets from the specified network device and receive queue
- rps_cpus files use comma-delimited CPU bitmaps
- To allow a CPU to handle interrupts for the receive queue on an interface, set the value of their positions in the bitmap to 1

== m10p18_network_ref_config_rps_1

Receive Packet Steering (RPS) is similar to RSS in that it is used to direct packets to specific CPUs for processing. However, RPS is implemented at the software level, and helps to prevent the hardware queue of a single network interface card from becoming a bottleneck in network traffic.

RPS has several advantages over hardware-based RSS:

RPS can be used with any network interface card.
It is easy to add software filters to RPS to deal with new protocols.
RPS does not increase the hardware interrupt rate of the network device. However, it does introduce inter-processor interrupts.

RPS is configured per network device and receive queue, in the /sys/class/net/device/queues/rx-queue/rps_cpus file, where device is the name of the network device (such as eth0) and rx-queue is the name of the appropriate receive queue (such as rx-0).

The default value of the rps_cpus file is 0. This disables RPS, so the CPU that handles the network interrupt also processes the packet.

To enable RPS, configure the appropriate rps_cpus file with the CPUs that should process packets from the specified network device and receive queue.

The rps_cpus files use comma-delimited CPU bitmaps. Therefore, to allow a CPU to handle interrupts for the receive queue on an interface, set the value of their positions in the bitmap to 1. For example, to handle interrupts with CPUs 0, 1, 2, and 3, set the value of rps_cpus to 00001111 (1+2+4+8), or f (the hexadecimal value for 15).

Network Performance Configuration

For network devices with single transmit queues, configure RPS to use CPUs in the same memory domain
- On non-NUMA systems, this means all available CPUs can be used
- If network interrupt rate is extremely high, exclude the CPU that handles network interrupts to improve performance
For network devices with multiple queues, there is typically no benefit to configure both RPS and RSS
- RSS is configured to map a CPU to each receive queue by default
- RPS may still be beneficial if there are fewer hardware queues than CPUs
- RPS is configured to use CPUs in same memory domain

Network Performance Configuration

Configuring Receive Flow Steering (RFS)

RFS is disabled by default. To enable RFS, you must edit two files:

/proc/sys/net/core/rps_sock_flow_entries
- Set the value to the maximum expected number of concurrently active connections.
- Red Hat recommends a value of 32768 for moderate server loads.
- All values are rounded up to the nearest power of 2.
/sys/class/net/device/queues/rx-queue/rps_flow_cnt
- device = name of the network device you wish to configure (for example, eth0).
- rx-queue = receive queue you wish to configure (for example, rx-0).
- Set the value to rps_sock_flow_entries divided by N.
  - N = number of receive queues on a device.
  - For single-queue devices, the value of rps_flow_cnt is the same as the value of rps_sock_flow_entries.
If data received from a single sender is greater than a single CPU can handle, do one of the following:
- Configure a larger frame size
- Consider NIC offload options
- Consider faster CPUs

Consider using numactl or taskset in conjunction with RFS to pin applications to specific cores, sockets, or NUMA nodes. This can help prevent packets from being processed out of order.

== m10p20_network_ref_config_rfs

Receive Flow Steering (RFS) extends RPS behavior to increase the CPU cache hit rate and thereby reduce network latency. Where RPS forwards packets based solely on queue length, RFS uses the RPS backend to calculate the most appropriate CPU, and then forwards packets based on the location of the application consuming the packet. This increases CPU cache efficiency.

RFS is disabled by default. To enable RFS, you must edit two files:

/proc/sys/net/core/rps_sock_flow_entries
Set the value of this file to the maximum expected number of concurrently active connections. We recommend a value of 32768 for moderate server loads. All values entered are rounded up to the nearest power of 2, in practice.
/sys/class/net/device/queues/rx-queue/rps_flow_cnt
Replace device with the name of the network device you wish to configure (for example, eth0), and rx-queue with the receive queue you wish to configure (for example, rx-0).

Set the value of this file to the value of rps_sock_flow_entries divided by N, where N is the number of receive queues on a device. For example, if rps_flow_entries is set to 32768 and there are 16 configured receive queues, rps_flow_cnt should be set to 2048. For single-queue devices, the value of rps_flow_cnt is the same as the value of rps_sock_flow_entries.

Data received from a single sender is not sent to more than one CPU. If the amount of data received from a single sender is greater than a single CPU can handle, configure a larger frame size to reduce the number of interrupts and therefore the amount of processing work for the CPU. Alternatively, consider NIC offload options or faster CPUs.

Consider using numactl or taskset in conjunction with RFS to pin applications to specific cores, sockets, or NUMA nodes. This can help prevent packets from being processed out of order.

Network Performance Configuration

Configuring Accelerated RFS

Accelerated RFS is only available if the following conditions are met:
- The NIC must support accelerated RFS by exporting the ndo_rx_flow_steer() netdevice function.
- ntuple filtering must be enabled.
Once these conditions are met, CPU to queue mapping is deduced automatically based on IRQ affinities configured by the driver for each receive queue (traditional RFS configuration).
Red Hat recommends using accelerated RFS wherever using RFS is appropriate and the NIC supports it.

Reference

Configuring Receive Flow Steering (RFS)

== m10p21_network_ref_config_accel_rfs

Accelerated RFS boosts the speed of RFS by adding hardware assistance. Like RFS, packets are forwarded based on the location of the application consuming the packet. Unlike traditional RFS, however, packets are sent directly to a CPU that is local to the thread consuming the data: either the CPU that is executing the application, or a CPU local to that CPU in the cache hierarchy.

Accelerated RFS is only available if the following conditions are met:

Accelerated RFS must be supported by the network interface card. Accelerated RFS is supported by cards that export the ndo_rx_flow_steer() netdevice function.
ntuple filtering must be enabled.
Once these conditions are met, CPU to queue mapping is deduced automatically based on traditional RFS configuration. That is, CPU to queue mapping is deduced based on the IRQ affinities configured by the driver for each receive queue.

Red Hat recommends using accelerated RFS wherever using RFS is appropriate and the network interface card supports hardware acceleration.

MTU and Jumbo Frames

MTU = Maximum Transmission Unit.
- Default MTU is 1500.
Jumbo frames = Frames larger than 1500.
Set an MTU as close to 9000 as possible.
- Allows for the best throughput on gigabit or better networks.
- Not all interfaces support MTU’s over 1500.
- Some interfaces support MTU’s over 1500, but not up to 9000.
All network gear in between, and both hosts, must support and be configured to the same MTU size.
- Network gear includes switches (VLANS), routers, and firewalls.
  If you change MTU values for a host, do it from the console, in case network connectivity to the host is interrupted.

MTU and Jumbo Frames

Use ip to find the maximum supported MTU size for a network interface:
```
[root@server1 ~]# ip link set eth0 mtu 9000
```
- If the ip command fails, the MTU value is not supported.
- To find the common supported value, try MTU values on all hosts that communicate with each other.
- Use jumbo frames on LANs only; most WAN network gear cannot support it.
- Using jumbo frames over tunneled connections is not recommended.
To configure a persistent MTU setting on an interface, add MTU=value to the ifcfg-interface file in /etc/sysconfig/network-scripts.
- value = the MTU value
- interface = the interface you are configuring

MTU and Jumbo Frames

Encryption Overhead and PMTU Issues

Encrypted tunnels between network segments (VPN, GRE, IPSec), may cause fragmented packets.
- If a TCP session starts normally over a tunneled link but goes down shortly after, this may be a symptom of fragmentation due to tunnel overhead.
- This is typical with SCP sessions.
Path MTU Discovery (PMTU) can automatically find the correct MTU to use over network segments. ** Dynamic PMTU discovery requires ICMP packet type 3 code 4.
- Many network and system administrators block all ICMP packets.
Possible solutions to PMTU issues:
- Enable ICMP type 3 code 4 between the two hosts and all network gear in between.
- Manually discover the necessary MTU value and set it on both hosts’ network interfaces:
  1. Disallow fragments (-M do) with ping and start with a packet size of 1472 (-s 1472).
    [root@server1 ~]# ping 192.168.1.6 -M do -s 1472 -c 1 PING 192.168.1.6 (192.168.1.6) 1472(1500) bytes of data. From 192.168.1.203 icmp_seq=1 Frag needed and DF set (mtu = 1500)
  2. Subtract one from the packet size until it works (-c 1 sends one packet):
    [root@server1 ~]# ping 192.168.1.6 -M do -s 1471 -c 1 PING 192.168.1.6 (192.168.1.6) 1471(1499) bytes of data. 1479 bytes from 192.168.1.6: icmp_seq=1 ttl=64 time=0.913 ms
  3. To determine a valid MTU size, add 28 to the packet size in the ping command that works.

== m10p24_mtu_jf_encrypt_overhead

In environments where encrypted tunnels are used between network segments (VPN, GRE, IPSec), packets may be fragmented due to encryption overhead. VPNs are becoming more common in today’s networks so this problem is becoming more common. When TCP sessions are started normally over a tunneled link but the sessions go down after a short period, this may be a symptom of fragmentation due to tunnel overhead.

This is typical with SCP sessions.

You can often use Path MTU Discovery (PMTU) to automatically find the correct MTU to use over network segments. Many network and system administrators block all ICMP packets for good reason, however ICMP packet type 3 code 4 are required for dynamic PMTU discovery to work correctly.

There are two possible solutions to PMTU issues:

Enable ICMP type 3 code 4 between the two hosts and all network gear in between.
Manually discover the necessary MTU value and set it on both hosts’ network interfaces.
To do this, disallow fragments (-M do) with ping and start with a packet size of 1472 (-s 1472), and then subtract one from the packet size until it works (-c 1 sends one packet):

To determine a valid MTU size, add 28 to the packet size in the ping command that works.

iSCSI Troubleshooting

iSCSI initiator: a client that needs access to raw SAN storage.
iSCSI target: a remote hard disk presented from an iSCSI server or "target portal".
iSCSI target portal: a server that provides targets over the network to an initiator
IQN (iSCSI Qualified Name): Each initiator and target needs a unique name to identify it
- Use a name that is likely to be unique on the Internet.
- Name format: iqn.yyyy-mm.{reverse domain}:label.

Network communication by default is cleartext to port 3260/tcp on the iSCSI target.

If you allow two initiators to log in to the same iSCSI target (remote hard disk) at the same time, it is important not to allow both initiators to mount the same file system from the same target at the same time. Unless a cluster file system such as GFS2 is in use, you risk file system corruption.

== m10p25_iscsi_ts

In this section you will learn to troubleshoot the use of iSCSI file systems.

iSCSI (Internet SCSI) supports sending SCSI commands from clients (initiators) over IP to SCSI storage devices (targets) on remote servers. An iSCSI Qualified Name is used to identify initiators and targets and follows the format of: iqn.yyyy-mm.{reverse domain}:label. Network communication by default is cleartext to port 3260/tcp on the iSCSI target.

iSCSI initiator: a client that needs access to raw SAN storage
iSCSI target: a remote hard disk presented from an iSCSI server, or "target portal"
iSCSI target portal: a server that provides targets over the network to an initiator
IQN: "iSCSI Qualified Name". Each initiator and target needs a unique name to identify it; best practice is to use one likely to be unique on the Internet.

iSCSI Troubleshooting

iSCSI Initiator Review

Red Hat Enterprise Linux typically uses a software-based iSCSI initiator.
- Software-based iSCSI initiators must connect to an existing Ethernet network with bandwidth that supports expected storage traffic.
- Hardware-based iSCSI initiators include the required protocols in a dedicated host bus adapter (HBA).
- iSCSI HBAs and TCP offload engines (TOE), which include the TCP network stack on an Ethernet NIC, move processing overhead and Ethernet interrupts to hardware.
Configuring an iSCSI client initiator requires installing iscsi-initiator-utils
- Includes iscsi and iscsid services.
- Includes /etc/iscsi/iscsid.conf and /etc/iscsi/initiatorname.iscsi configuration files.
As an iSCSI node, the client requires a unique IQN.
- /etc/iscsi/initiatorname.iscsi contains a generated IQN using Red Hat’s domain.
- Administrators typically reset the IQN to their own domain and client system string.
/etc/iscsi/iscsid.conf contains default settings for node records created during new target discovery.
- Settings include iSCSI timeouts, retry parameters, and authentication usernames and passwords.
- Changing this file requires restarting iscsi.
  [root@desktop1 ~]# systemctl restart iscsi

== m10p26_iscsi_ts_initiator_rvw_1

In Red Hat Enterprise Linux; an iSCSI initiator is typically implemented in software, and functions similar to a hardware iSCSI HBA to access targets from a remote storage server. Using a software-based iSCSI initiator requires connecting to an existing Ethernet network of sufficient bandwidth to carry the expected storage traffic.

iSCSI can also be implemented using a hardware initiator that includes the required protocols in a dedicated host bus adapter. iSCSI HBAs and TCP offload engines (TOE), which include the TCP network stack on an Ethernet NIC, move the processing of iSCSI or TCP overhead and Ethernet interrupts to hardware, easing the load on system CPUs.

Configuring an iSCSI client initiator requires installing the iscsi-initiator-utils package, which includes the iscsi and iscsid services and the /etc/iscsi/iscsid.conf and /etc/iscsi/initiatorname.iscsi configuration files.

As an iSCSI node, the client requires a unique IQN. The default /etc/iscsi/initiatorname.iscsi file contains a generated IQN using Red Hat’s domain. Administrators typically reset the IQN to their own domain and an appropriate client system string.

The /etc/iscsi/iscsid.conf file contains default settings for node records created during new target discovery. Settings include iSCSI timeouts, retry parameters, and authentication usernames and passwords. Changing this file requires restarting the iscsi service.

iSCSI Troubleshooting

iSCSI Initiator Review

To discover targets, install iscsi-initiator-utils, and then enable and start iscsi.
- Targets must be discovered before device connection and use.
- Discovery process stores target node information and settings in /var/lib/iscsi/nodes and uses defaults from /etc/iscsi/iscsid.conf.
- Node records are stored for each portal.
Perform discovery with this command:
```
[root@desktop1 ~]# iscsiadm -m discovery -t sendtargets -p target_server[:port]

192.168.0.101:3260,1 iqn.2014-06.com.example:server1
```
- In discovery mode, sendtargets returns only targets with access configured for this initiator.
- Omit port number if target server is configured on default port 3260.
- Upon discovery, a node record is written to /var/lib/iscsi/nodes and used for subsequent logins.

iSCSI Troubleshooting

To use the listed target, log in using this command:
```
[root@desktop1 ~]# iscsiadm -m node -T iqn.2014-06.com.example:server1 [-p target_server[:port]] -l
```
- Specifying the portal is optional.
- A login without specifying a portal connects to every portal node that accepts this target name.
Once discovered, obtain information about targets with iscsiadm
- Set the command detail level with -P N, where N is the verbosity setting. 0 specifies the least verbose output.
- Show information about discovered targets: iscsiadm -m discovery [-P 0|1]
- Show information about known targets: iscsiadm -m node [-P 0|1]
- Show information about active sessions: iscsiadm -m session [-P 0|1|2|3]
To discontinue using a target temporarily, log out.
- Node records remain after logout and are used to automatically log into targets upon system reboot or iscsi restart.
- Log out of a target using this command:
  [root@desktop1 ~]# iscsiadm -m node -T iqn.2012-04.com.example:example [-p target_server[:port]] -u
- If a portal is not specified, the target logs out of all relevant portals.

== m10p28_iscsi_ts_initiator_rvw_3

To use the listed target, log in using the following command:

Specifying the portal is optional. If the target exists on multiple portals (e.g., in a multipathed, redundant server configuration), performing a login without specifying a portal will connect to every portal node that accepts this target name.

Once discovered, obtain information about targets with the iscsiadm command. Use the option -P N to set the command detail level, with 0 specifying the least verbose output.

To discontinue using a target, use iscsiadm to log out temporarily. By design, node records remain after logout and are used to automatically log into targets upon system reboot or iscsi service restart. Log out of a target using the following command (notice the similarity to login):

If a portal is not specified, the target logs out of all relevant portals. To log into the target again, repeating discovery is not necessary since the node records already exist.

iSCSI Troubleshooting

To permanently log out of a target, delete the node records.
- After deletion, manual or automatic login cannot reoccur without performing another discovery.
- Not specifying a portal removes target node records for all relevant portals.
- Delete the node record permanently by using this command:
  [root@desktop1 ~]# iscsiadm -m node -T iqn.2012-04.com.example:example [-p target_server[:port]] -o delete

To list iSCSI targets:

[root@desktop1 ~]# dmesg|tail
...OMITTED...
[ 5153.388625]  sda: unknown partition table
[ 5153.403314] sd 8:0:0:0: [sda] Attached SCSI disk
[root@desktop1 ~]# lsblk
NAME                       MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0                          2:0    1    4K  0 disk
sda                          8:0    0  100M  0 disk
...OMITTED...
[root@desktop1 ~]# tail /var/log/messages
...OMITTED...
Dec  3 13:55:31 desktop1 kernel: sda: unknown partition table
Dec  3 13:55:31 desktop1 kernel: sd 2:0:0:0: [sda] Attached SCSI disk
Dec  3 13:55:31 desktop1 iscsid: Connection1:0 to [target: iqn.2014-06.com.example:server1, portal: 192.168.0.101,3260] through [iface: default] is operational now

iSCSI Troubleshooting

List the targets recognized by the iscsi service.

[root@desktop1 ~]# iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 6.2.0.873-21
Target: iqn.2010-09.com.example:rdisks.desktop1 (non-flash)
        Current Portal: 192.168.0.254:3260,1
        Persistent Portal: 192.168.0.254:3260,1
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.1994-05.com.redhat:ffb991f28cb
                Iface IPaddress: 192.168.0.1
                Iface HWaddress: <empty>
                Iface Netdev: %lt;empty>
                SID: 7
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 120
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 262144
                FirstBurstLength: 65536
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 8  State: running
                scsi8 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sda          State: running

iSCSI Troubleshooting

Change directory to the location of the iSCSI node records for the remainder of this exercise. Locate the persistent node record for the new iSCSI target.
```
[root@desktop1 ~]# cd /var/lib/iscsi/nodes
[root@desktop1 nodes]$ ls -lR
```

View the connection parameters defaults in an iSCSI node record.

[root@desktop1 nodes]# less iqn.2010-09.com.example\:rdisks.desktop1/192.168.0.254\,3260\,1/default

When persistently mounting a file system on an iSCSI target in /etc/fstab:

Use blkid to determine the file system UUID and mount using UUID, not /dev/sd* device name.
Use _netdev as a mount option in /etc/fstab.
Ensure the iscsi and iscsid services will start at boot time.

References

Online Storage Management
/usr/share/doc/iscsi-initiator-utils-*/README
Can I put a swap device or file on iSCSI storage?
iscsiadm(8) and iscsid(8) man pages
Open-iSCSI

== m10p31_iscsi_ts_initiator_rvw_6

Change directory to the location of the iSCSI node records for the remainder of this exercise. Locate the persistent node record for the new iSCSI target.
View the connection parameters defaults in an iSCSI node record.

When persistently mounting a file system on an iSCSI target in /etc/fstab:

Use blkid to determine the file system UUID and mount using UUID, not /dev/sd* device name. (The device name can come up differently from boot to boot depending on the order in which iSCSI devices respond over the network. This can cause the wrong device to be used if mounting by device name.)
Use _netdev as a mount option in /etc/fstab. (This ensures that the client will not attempt to mount the file system until networking is up. Otherwise the system will have errors at boot.)
Ensure the iscsi and iscsid services will start at boot time.

Module Completion

Nice job!

Click the button below to complete this module of the course: